Dynamics of Random Networks: Connectivity and First Order 

Phase Transitions 



Albert-Laszlo Barabasi 
Department of Physics, University of Notre Dame, Notre Dame, IN 46556. 

(February 1, 2008) 

Abstract 

The connectivity of individual neurons of large neural networks determine 
both the steady state activity of the network and its answer to external stim- 
ulus. Highly diluted random networks have zero activity. We show that 
increasing the network connectivity the activity changes discontinuously from 
zero to a finite value as a critical value in the connectivity is reached. Theo- 
retical arguments and extensive numerical simulations indicate that the origin 
of this discontinuity in the activity of random networks is a first order phase 
transition from an inactive to an active state as the connectivity of the net- 
work is increased. 
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Networks of neuron type threshold elements have generated a lot of interest lately, mo- 
tivated by their potential for reproducing neurobiological processes and understanding the 
generic mechanism governing basic brain functions. Most of the studied models deal either 
with fully connected networks (for which every neuron is connected to all neurons in the 
system [|I|) or highly diluted networks [^^. At this point little is known about the properties 
of networks with arbitrary connectivity, including networks whose connectivity is realistic 
from the standpoint of biology. 

Here we take a first step towards understanding the dynamical properties of random 
networks by investigating the effect of connectivity on the network dynamics |Q, where 
the connectivity is the probability that two randomly selected neurons have a synaptic 
connection. A network with small connectivity is inactive, i.e. if we excite it with an 
external stimulus, the activity dies out in a short time. One would expect that increasing 
the connectivity leads to a gradual increase in the activity of the network. Contrary to this 
simple picture, here we show that upon increasing the connectivity, the steady state activity 
does not increase continuously, but jumps discontinuously, the network going through a 
first order phase transition from an inactive to an active state. This result may help us 
understand the evolutionary driving forces that lead to the synaptic densities characterizing 
the brain by providing the range of connectivities necessary for the network to display a 
nonzero activity, and sheds light into the difficulties occurring during modeling of low level 
sustained activity in the cerebral cortex |Q. 

Consider a network formed by neurons. Each neuron is connected randomly to any of 
the other neurons with a probability p, i.e. p is measure of the network connectivity. The 
detailed wiring diagram of the system is given by the matrix J, whose element Jij is 1, if 
there is a directed path from neuron i to j, and zero otherwise. 

First we consider a simplified version of the McCulloch-Pitts model [^], limiting the 
model to only excitatory neurons. Later we will show that this simplification does not 
affect the main conclusions of the paper. Every neuron i can be in two possible states, 
active (sj = 1) or inactive (sj = 0). Using synchronous updating, the state of neuron 
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i at time t + 1 is Si{t + 1) = 1 with probability P{hi,T) = [1 + tanh(/ij(t)/T)]/2, and 
+ 1) = otherwise. Here hi{t) = J2j JjiSj{t) — 6, 6 is the threshold of the individual 
neuron, and T (temperature) characterizes the spontaneous firing of a neuron. The activity 
of the network can be characterized by the normalized number of active neurons at time t: 
x{t) = {l/N)J2iSiit)- Hereafter we call a network active (inactive) if x(t) ^ {x(t) = 0), 
for large t. 

After activating a fraction x of all neurons, if the network is highly diluted (p 0), 
the neurons are not connected to sufficient number of other neurons to overcome 6, and the 
activity decays to zero. In contrast, a highly connected network (p — > 1) for almost any 
initial conditions leads to an active state. The steady state dynamics in the active state is 
rather complex, leading to periodic oscillations at T = 0, whose period depends on A^, p 
and J. However, at nonzero temperatures the dynamics is quasiperiodic, i.e. the system 
explores randomly a number of periodic orbits. 

Our goal is to understand the changes in the network activity as the connectivity p 
is varied. For this we implement a numerical method to calculate the free energy, that 
characterizes the dynamics of networks with fixed {N,p). For now we limit ourselves to 
symmetric networks, for which Jij = Jji. 

Consider the above network model after it reached its steady state and define f'{x) as 



Here f'{x) depends on J and on the initial conditions {xi{t = 0)}. Averaging over the various 
realizations of the network topology, J, leads to /(x) = [f'{x)]j^x(o), which is a smooth and 
univalued function. Alternatively, we can define the continuum version of (JlD, using the 
averaged /(x) 



where we added the uncorrelated noise, rjit), to incorporate the randomness of the updating 
rule at nonzero temperatures. 




(1) 



dtx{t) = f{x{t)) + r]{t) 



(2) 
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Formally, can be generated from the free energy T{x) using 

dtx{t) = -^^+r^{t), (3) 

where 

J^[x) = - £ f{x')dx'. (4) 

As we show below, using (|^) and (Q), can be calculated using a mean field approximation 
and measured in numerical simulations. 

Mean field theory (MFT) — At time t we randomly activate a fraction x{t) of neurons. 
The goal is to calculate x{t + 1). The calculation proceeds in three steps. 

(i) Choosing randomly a neuron, the probability P{N,k,p) that it is connected to k 
other neurons follows the binomial distribution P{N,k,p) = (^'^^^^p^{l — p)^~^~^, where 
(:)=a!/6!(a-6)!. 

(ii) Knowing that a fraction x of all neurons are active, the probability n(fc,m,x), that 
of the k neurons the chosen neuron is connected to, m are active, is given by the binomial 
distribution n(/c,m, x) = (^^x'^{l — x)^^^. (iii) Finally, a neuron with m active neighbors 
is activated with probability P{m,T). 

Thus, the activity of the network, x{t + 1), is given by 

N-l N-1 

x{t + 1) = E T) E P(Ar, k, p)n{k, m, x{t)). (5) 

m=0 k=m 

This expression provides /(x) = x{t + 1) — and using (Q), we can calculate the free 
energy jF(x). 

Obtaining (|^) we made two approximations. In (i) we neglected the quenched nature of 
the randomness in the connectivity matrix J, replacing J with a new one at every time step, 
subject to fixed p and A^. In (ii) we assumed that every neuron has the same probability 
X to be active, independent of the network topology. In reality the activity of a neuron 
is highly correlated with its connectivity. However, even within these approximations, the 
MFT captures correctly the nature of the phase transition from the inactive to the active 
state. 
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Phase transition — (a) The free energies obtained from (|]) for T = 0, N = 100, 6 = 2, 
and various values of the connectivity p are shown in Fig. |l](a). For small p the only stable 
state has zero activity {x = 0), i.e. starting with any x, the activity decays. Increasing 
p, at pi(0) ~ 0.042 the free energy develops an inflection point, leading to a second, local 
and unstable minima at Xi(0), corresponding to a nonzero activity. Further increasing p, at 
Pc{0) ~ 0.046, J^{xi) becomes smaller than jF(x = 0), indicating that the system undergoes 
a first order phase transition during which the activity jumps from x = to Xi 7^ 0. The 
formally stable x = state becomes a metastable state. 

(b) For nonzero T, thermal fluctuations induce a thermal activity, xo{T) = [1 + 
tanh(— 0/T)]/2, that is independent of the network topology. Expanding for small p 
and X, we find that the derivative of the free energy in x = is negative for nonzero T, 
thus JF has no minima at x = 0, but only at the thermally activated Xq{T). For small T, 
increasing p leads to the appearance of the metastable xi(T) > xo(r), which becomes a 
global minima at Pc{T) (see Fig. |l|(b)). Thus for small T we observe a first order phase 
transition from the thermally activated state, xq{T), to the active state Xi(T). 

(c) For high T the thermally induced activity, Xo(r), dominates the dynamics of the 
system, such that for T > the local minima corresponding to the active state Xi (T) does 
not appear, and we can not distinguish between the thermally activated state and the active 
state of the network (see Fig. |l|(c)). 

We summarize the above behavior using the (p, T) phase diagram, shown in Fig. [^(d). 
For small T the active and the thermally activated states are separated by the line Pc{T). 
However, the Pc(T) line ends at T^, and for T > Tc the system is too noisy to distinguish 
the thermally activated state from the true activity of the network. 

Steady state behavior — Due to the discussed approximations, the MFT does not provide 
us the steady state behavior of the network. For this we measured the steady state /(x) 
using the discussed network model. Fig. ^ shows the measured free energy for T = and 
for T > Tc. For T = we observe a first order phase transition from the inactive to the 
active states, as predicted by the MFT (see Fig. ||(a)). For T 7^ 0, thermal fluctuations lead 

5 



to a thermally induced metastable state , which is indistinguishable from the active state 
of the network if T > T^. Thus the phase diagram for the steady state behavior is similar 
to the one predicted by the MFT, shown in Fig. |l|(d). 

Both the MFT and simulations in the steady state predict the existence of a first order 
phase transition for small temperatures as the connectivity of the network is increased. A 
number of important questions arise at this point [§] : How generic is the observed first order 
phase transition? What are the implications of this transition? 

To answer the first question, we studied extensions of the described model, including 
elements that are often considered in neural network simulations. Simulations show that 
introducing asymmetry ( Jj^ 7^ Jji) modifies the Pc(T) curve, but the system still undergoes a 
first order phase transition. We find that the phase transition is not affected by the inclusion 
of inhibitory neurons either, nor by the nature of updating process (random or synchronous). 
Thus we conclude that the existence of a first order phase transition for small temperatures 
is a generic property of random networks, and it is related to the topology of the network 
rather than the dynamics of the individual neurons. 

What determines the actual connectivity and the synaptic density of certain parts of 
the brain? Naturally, a complete answer to this question should consider the genetically 
determined non-randomness in the synaptic connections. However, our results suggest that 
for random networks the optimal connectivity depends on the dynamical properties desired 
for the particular network. When a certain fraction of neurons are activated by an external 
stimulus, the network should respond with nonzero activity. For most brain functions this 
activity should decay after some time if the stimulus is not sustained. For a large network 
this can be achieved by exciting the network into a metastable state. In this case the activity 
decays after some characteristic time, which depends on the free energy barrier between the 
stable and the metastable states. On the other hand, if sustained activity is the goal, one 
needs to use a p larger than Pc{T). For a long time cerebral cortex modeling was halted by 
the inability of large random network models to reproduce the observed low level activities 

The observed first order phase transition explains the origins of this failure: the arbitrary 
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small activity between zero and x{pc{T)) is simply not available in network models due to 
the jump in the activity at the transition point. 

Furthermore, the outlined method (|1]-^ could be used in exploring the properties of large 
networks when only a small fraction of the neurons can be monitored in the laboratory. 
Examples include multi-electrode recording techniques, that provide information about the 
activity of typically tens of neural cells Such measurements may help understand the 
network activity by providing jF(x) for small x. 
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FIGURES 

FIG. 1. The free energy predicted by the mean-field theory (|5|). (a) T = 0. The curves, from 
top to bottom, correspond to p = 0.020,0.042,0.046,0.050. (b) T = 1.11. The curves, from top to 
bottom, correspond to p = 0.020, 0.042, 0.043, 0.045. (c) T = 10. The curves, from top to bottom, 
correspond to p = 0,0.020,0.070,0.101. (d) The (T,p) phase diagram. The solid line corresponds 
to Pc{T), separating the thermally activated phase from the active state. The error bars are smaller 
than the symbols. 

FIG. 2. The steady state free energy obtained from numerical simulations for = 100 and 
9 = 2. Average over 1000 runs is taken, (a) T = 0. The curves, from top to bottom correspond 
to p = 0.005,0.010,0.011,0.015. (b) T = 10.0. The curves, from top to bottom, correspond to 
p = 0.0,0.030,0.043,0.101. 
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